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ABSTRACT 



As one of several efforts to address the low 



retention rates of beginning teachers in California schools, the 
Beginning Teacher Support and Assessment Project (BTSA) is providing 
support for new teachers along with a component that assesses their 
classroom competencies. The assessment component and planned 
evaluation studies are described. BTSA will use the Educational 
Testing Service's (ETS) Praxis III assessment program, which is 
designed to provide data on 19 criteria clustered into 4 domains of 
teacher competence. Trained assessors collect data from teacher 
questionnaires, classroom observation, and interviews. This data is 
sent to ETS for processing and summary. Beginning teachers in BTSA 
would be linked with a support provider who is an experienced teacher 
and who would give assistance but would not evaluate performance. Of 
the approximately 165 beginning teachers in the 5 participating urban 
school districts, 45 were selected as the experimental group. Data 
will be analyzed to examine professional growth variables and 
beginning teacher perceptions. The large amounts of data collected 
will yield information on professional growth that has important 
implications for teacher retention and educational improvemen\: • 
(Contains 18 references.) (SLD) 
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Background 

^ In 1988, the California State Legislature appropriated funds for a pilot project entitled the 

^ California New Teacher Project (CNTP). As its main purpose, the CNTP addressed the low 
Q retention rate of beginning teachers in California schools. Numerous studies had reponed that more 
than a quarter of beginning teachers left the profession during the first five years. New teachers 
abandoned teaching in even greater numbers from urban, inner city schools (McLaughlin, et al., 
1986; Estes, et al, 1990). 

During its four years of operation, CNTP's 37 funded projects throughout the state reported 
dramatic increases in teacher retention. Colben and Wolff directed two separate CNTP programs 
from 1988-92. The CNTP incorporated a variety of models for structuring support for new 
teachers and averaged a 97% retention rate statewide. Research which emerged from the projects 
confirmed that retention and success of new teachers was directly related to the projects' intensive 
oite-based programs of support (Colbert and Wolff, 1992; Garmston and Banell, 1991; Ward and 
Dianda, 1990). 

In 1992, responding to these impressive data and concerned over the continuing difficulty in 
attracting and retaining qualified beginning teachers, the California State Legislature increased its 
commitment to beginning teacher suppoit and assessment by passing SB 1422 (Bergeson), 
providing approximately $4.9 million for the Beginning Teacher Support and Assessment Project 
^ (BTSA). In the spring of 1993, the Beginning Teacher Support and Assessment Project funded 15 
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sites throughout the state. A second cohort of 15 additional projects was added in the winter of 
1994. Plans call for continued expansion of BTSA funding to provide support to all beginning 
teachers throughout the state in coming years. 

While both the OSTTP and BTSA clearly share similar objectives, there is at least one notable 
difference in their mandates. Whereas the CNTP projects were focused on providing support for 
beginning teachers, the BTSA projects are required to include a wellndefined assessment 
component, in addition to beginning teacher support. In fact, roughly half of the funds available to 
each project was to be used to assess participating beginning . "achers on classroom competencies 
identified by research supported by the state (Stansbury and Long, 1992). The funding agencies 
required that assessment serve as another level of support for beginning teachers (as opposed to 
using assessment in the traditional, i.e., evaluative, sense). 

To assist applicants in meeting the assessment requirements, the California Commission on 
Teacher Credentialing and the Department of Education held workshops at which assessment 
options were discussed. At one of these sessions, three organizations which had responded to the 
state's call for proposals for classroom assessment procedures, made presentations. Applicants 
were informed that they could choose to work with one of these organizations or develop their own 
assessment component, as long as it addressed the guidelines established by the state. 

After consideration, we at the BTSA project at California State University, Dominguez Hills 
(CSUDH), chose to use the Educational Testing Service's (ETS) Praxis III assessment program. 
As presented at the state-sponsored meetings and in ETS literature. Praxis III had been carefully 
aligned with the teaching domains identified by previous research as important for tlie competence 
of all teachers and had been developed and tested in multicultural settings (Villegas, 1990). Such 
an assessment instrument seemed appropriate for the highly diverse and challenging classrooms in 
the south Los Angeles County area served by CSUDH. 

The CSUDH project is in collaboration with five urban school districts in the greater Los 
Angeles area. The majority of the beginning teacher panicipants are teaching on an emergency 



credential, i.e., they have not had any preservice training. Most are currently enrolled in a 
credential program, earning their preliminary credential while teaching fullH:ime. 

The Praxis HI assessment system is designed to provide data on nineteen criteria clustered into 
four domains. The four domains are: organizing content knowledge for student learning, creating 
an environment for student learning, teaching for student learning, and teacher professionalism. 
Trained assessors collect data from three sources: two pre-observadon questionnaires completed 
by the beginning teacher, a carefully scripted classroom observation, and pre- and 
post-observation interviews with the beginning teacher. Following the observation process, the 
assessor reads through all of the data collected and assigns each piece of information to one of the 
criteria to which it relates. Using these data, the assessor writes a summary statement that reflects 
the teacher's performance on the criterion and assigns a rating. The collected and categorized data, 
summary statements, and ratings are recorded and sent to the ETS, where they are processed. Both 
the beginning teacher and the BTS A staff receive copies of the summary statements for each of the 
nineteen criteria. ETS also provides additional numerical summary information to thie BTSA staff. 

Preliminary Concerns 

In the fall of 1993, initial comments from beginning teachers and assessors alened the BTSA 
staff to possible problems with the assessment process. In the first few months of the project, a 
number of beginning teachers withdrew from BTSA, citing the anxiety created by the assessment 
for their decision. Anecdotes from assessors also hinted at broad-based apprehension among the 
beginning teachers towards the assessors* classroom observations. The staffs concern was 
heightened after the administration of a questionnaire to all beginning teachers following the first of 
the three classroom assessments. A large number of these beginning teachers cited as problematic: 
the amount of time needed to complete the assessment forms prior to the classroom observation 
visit; ihe stress and anxiety of being assessed; and, the lack of timely feedback. 
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F -or to this, in informal staff discussions, we began to raise questions about the validity of 
using a highly structured assessment program as a suppon mechanism for first-year teachers. 
Another area of concern for us was the number of assessments that would be most appropriate for 
our purpose, i.e., using Praxis HI as another layer of support. Frankly, staff were concerned 
about the notion of using a structured assessment procedure in a context other than evaluation, i.e., 
as a support mechanism, because, no matter how we presented assessment or called it by any other 
name, the concept of assessment means evaluation to most people and often results in stress and 
anxiety for participants. At about the same time, events pushed us to make an important decision. 

During the early fall of 1993, three of our trained assessors withdrew from the program for a 
variety of personal reasons. Faced with the options of **borrowing" assessors from other projects 
which were as strapped for trained assessors as we were, or conducting additional assessments 
ourselves, we decided to assess one-quaner of the beginning teachers only once at the end of the 
yciic, instead of three times during the year, to compare the effects of multiple assessments versus a 
single assessment. 

Literature Review 

With this decision made, we began in earnest to examine the research literature on assessment 
and beginning teachers. Specifically, we were interested in exploring the role of assessment in 
assisting beginning teachers to improve their classroom teaching. 

As evaluation or assessment! of teachers seemed to be a common component of many induction 
programs (Schaffer, et 1992; Littleton, 1991; Carter, et al. 19^2^ we were surprised to find so 
little evidence of its benefits in the suppon of beginning teachers. Only Moore (1990) clearly saw 
the evaluation component as having noticeable positive effects, reporting that the majority of 
teachers in an induction program believed that the evaluation information they received from their 
mentors was helpful. 



A much more common theme centered upon the pitfalls of using assessment in induction 
programs. Successful induction of novice teachers into the profession seemed to be built on trust 
and concern among participants (Debolt, 1989). Assessment (usually founded upon hierarchical 
structures and unequal power bases) and mentoring (marked by approachability and 
encouragement) can be a volatile mix, requiring a careful and light hand (Heller and Sindelar, 
1991; Wasserman and Emery, 1989). Some researchers, such as Klug and Salzman (1990), noted 
that effective induction programs seem to emphasize assistance over evaluation, while others 
(Bas-Issac, 1989; Heller and Sindelar, 1991 ) warned that mixing evaluation and mentoring 
threatened to weaken or destroy the effectiveness of the induction process. Consisted with these 
findings, Hoffman and O'Neal (1985) found that their assessment component was seen as 
insignificant in helping teachers improve. Further, for some beginning teachers, it "caused 
irritation and even hostility" (p. 18). 

Berliner (1989) conducted research which provided a theoretical basis for our concerns. He 
described five theoretical stages of development that teachers progress through, while admitting, 
"we have only the scantiest knowledge about the ways that one progresses from novice to expert" 
(p. 39). Nevertlieless, **the evaluation of teachers also depends on such implicit theories of 
development.'* (p. 40). He continued, 

**What one chooses to observe or test for. when one expects to see it, how it should 
be measured, and the criteria by which successful performance is judged all depend 
on some notions, perhaps fragmentary, about the development of ability in 
pedagogy" (p. 40). 
He argued that 

"until extensive classroom experience has been acquired, there may be too littie 
in the minds of preservice teachers about what actions might be realistic, 
relevant, appropriate, moral, and so forth. Any analysis of teachers' actions and 
the practical arguments that they make to justify them, must be well 
contextualized, and thus requires extensive experience"(p. 64). 



The research literauire added weight to the concerns that we had begun to develop from our 
informal interactions with the participants in our program. Assessment of beginning teachers may 
create high levels of stress and anxiety for individuals already faced with a myriad of difficult new 
tasks and responsibilities. If, further, its implementation is both expensive and time-intensive, the 
rationale for its use would have to be compelling. The literature was markedly silent on the issue of 
changes in teachers' classrooms skills and competencies related to assessment. Our focus, then, 
became clearer: (1) to determine whether repeated structured assessment during the first year of 
teaching is valid as a support mechanism given Berliner's arguments and our own data; and, (2) to 
document the effects on beginning teachers' perceptions of the profession related to using a 
repeated and structured assessment process. 

Hypotheses 

There are two hypotheses: 

1. With intensive, structured, ongoing support for beginning teachers in place, repeated, 

structured assessment will have no significant effect on their professionaJ competence. 

2, With intensive, structured, ongoing support for beginning teachers in place, repeated, 

structured assessment will have no significant effect on their perceptions of the teaching 
profession. 

Research Design 

As with many research projects, our design emerged from and interacted with the realities of 
running a large support program. As originally conceptualized, a total of 180 beginning teachers 
would participate in the CSUDH BTSA project. Each of these Vginning teachers was linked with 
a "support provider", i.e., an experienced, nurturing teacher who would provide suppon and 
assistance, but not evaluation. In addition, the teachers would be observed three times over the 
course of the academic year by a trained assessor using the Praxis III instrument. 



Identification of beginning teachers started in the spring of 1993, with approximately 125 
beginning teachers identified anaong the five participating urban school districts. By October, the 
number of beginning teacher participants increased to approximately 165, With the decision to 
implement the research project, 45 of these beginning teachers (roughly twenty-five percent of the 
anticipated 180) were randomly selected as our experimental group. This group would only be 
assessed during the third cycle in May, foregoing both the fall and spring assessments. The 
remaining participants (an anticipated 135) were to be assessed three times. 

As the fall continued, . .dditional beginning teachers were recruited from the participating school 
districts, with other beginning teachers dropping from the program for various reasons. As these 
late-arrivers (21 beginning teachers) joined the program after the assessment process had begun 
and had not received orientation to the Praxis assessment process, the staff decided that they would 
be treated in a manner similar to the experimental group, with only the late spring assessment being 
carried out. 

With the number of participants seemingly stable, we count 187 beginning teachers. From this 
group, 121 (comparison group) will be assessed three times; 45 (experimental group one) were 
randomly chosen to be assessed only the third time, and the additional 21 (experimental group two) 
will receive only the third assessment because of their late enrollment in the program. For the 
purpose of analysis, experimental group two will be treated separately unless preliminary data^ 
analysis indicates that there are no differences between experimental groups one and two. 

To accommodate the range of school calendars, student-free days, natural disasters, and special 
events among the five districts and sixty-five schools participating in the program, assessors were 
given flexible target dates for completing the three assessments. The first assessment was to have 
been completed by the end of January, the second by the end of March, and the third by the end of 
May. During this third assessment, all 187 beginning teachers will be assessed. 
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Data Collection 

In attempting to determine the effects of the assessment process upon beginning teachers, we 
will draw on a number of data sources. First, scores for each beginning teacher on the Praxis III 
Assessment Institiment will be collected. For our comparison group of 121 beginning teachers, 
three separate scores for each of the nineteen criteria are expected. For the remaining beginning 
teachers, criteria scores will be collected only for the third and final assessmert cycle. We 
recognize a limitation to these data in that the scale used in Praxis III is truncated, i.e., from 1.0 to 
3.5 in 0.5 increments. 

Additional information is also being collected to gauge other possible effects of the assessment 
process. As part of the support component of the project, beginning teachers and support providers 
attend a number of professional development workshops throughout the year on a range of topics 
identified at an early stage of the project through a needs assessment. At each session, participants 
fill out evaluatiOi^s of the sessions that include open-^nded questions about on-going aspects of 
the project. 

Additional detailed information will also be collected from all project participants. In June, 
beginning teachers, support providers, and assessors will be asked to complete program 
questionnaire forms. Several questions on the form will request information abouc the effects of 
Praxis III upon professional skills and perceptions of the profession, school, and self. 

From these responses, important issues will be identified and an interview protocol developed. 
A random sample of beginning teachers and support providers will be selected to be interviewed. 
These interviews will permit an in-depth exploration of participants' perceptions of all program 
components. These interviews will be taped and transcribed for analysis. 

Data Analysis 

The data will be analyzed to examine professional growth variables and beginning teacher 
perceptions. The relationship of assessment to professional growth of beginning teachers as well 
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as tlieir perceptions about the process may be also be related to retention. 

We realize that we are collecting data on a large number of variables. These data will require 
quantitative and qualitative ajialysis as appropriate. Preliminary considerations have been discussed 
concerning the appropriate analysis tools. Professional growth variables, described below, will be 
analyzed using multivariate statistical procedures, such as multivariate analysis of variance and 
multiple regression. The second set of variables, beginning teacher perceptions, also described 
below, will be analyzed using content analysis procedures. 

Professional growth 

Data from the Praxis III instrument will allow us one measure of the growth of the beginning 
teachers' professional competence from the time they began in the program (spring, 1993 or fall, 
1994) to the end of the 1993-4 academic year. By examining differences in assessor ratings on 
each of the nineteen criteria from the first to the third assessment, we will develop a performance 
growth profile for each beginning teacher. Another measure will be support provider perceptions, 
as measured by questionnaire, of their beginning teacher's proficiency at the end of the year. 

More interesting data are likely to emerge from funher comparisons. As assessment is seen as 
another level of support within the context of the BTSA projects, we may see differences in 
pedagogical competence and/or teacher perceptions that contribute to teaching effectiveness and 
retention. Because our experimental group(s) of the beginning teachers will be assessed only at tiie 
end of the year, they will not have received this aspect of suppon. Using data collected from this 
single assessment, we will construct a similar performance profile for them. This profile will 
represent a "snapshot" of their professional competence on the nineteen criteria at the end of the 
year. As they did not participate in the year-long assessment process, these ratings would reflect 
only benefits from the support component. By comparing the ratings of these two groups, i.e., 
those who were assessed three times and those who were assessed once at the end, we will have a 

9 

10 



better understanding of the relationship between beginning teacher suppon and assessment, and the 
effect each has on retention, professional competence, and perceptions about the field. 

Beginning Teacher Perceptions 

We will use several other sources to analyze perceptions of beginning teachers. Questionnaires 
following the initial assessment, evaluations completed after xrzi?'^^ sessions, and 
end-of-the~year questionnaires will be used to construct a set of perceptions. Perceptions of those 
in the comparison group and those in the experimental group(s) will be compared. As an additional 
source of data, transcripts from the interviews will be analyzed and coded according to themes that 
emerge. 

Importance 

Assessment is big business. In the state BTSA projects alone, nearly $2.5 million were 
allocated for the first year for assessment purposes. As suggested in the literature, however, the 
nature of the benefits-if any-to beginning teachers is unclear. Further, assessment of beginning 
teacher performance may have detrimental effects on teachers' perceptions that may interfere with 
their successful induction into the profession. This study looks directly at these critical issues. Its 
findings may have important implications for construction of teacher induction programs. Funher, 
it may have powerful implications for the use of assessment with first year teachers for licensing or 
any other purpose. Finally, these findings may challenge unexamined assumptions about the role 
of beginning teacher assessment in improvement of California's schools. 
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Notes 

1. Vfliile the terms ''evaluation" and "assessment** are often given specific meanings in many 
contexts, we found that they seemed to be used interchangeably in most discussions of beginning 
teacher induction progr^ims. 
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